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Abstract 

This study used peer-reviewed published research reports to teach a seminar on learning 
and memory to first-semester college students. Complete reports (not summaries, re¬ 
views, or news reports) were re-written by this author to be more “student friendly” to 
college freshmen. These adapted published research reports (APRRs) retained original 
structure and key data but omitted references and tangential data while providing ex¬ 
planatory notes. 

Rather than lecturing the students about the papers, the approach was to engage students 
directly in scientific thinking by requiring them to work first individually and then as 
analysis teams to conduct a simulated peer review of the APRRs. To prod higher-level 
analysis, students were required to develop insightful answers to 21 scaffolding ques¬ 
tions. All questions required critical thinking, and seven specifically called for creative 
responses. The questions prompted the students to think deeply about alternative ap¬ 
proaches for testing, organizing and presenting results, meaning of the results, and the 
broader implications of the research. 

An end-of-course survey revealed that students regarded the experience positively. Most 
believed they learned more than they would have from lectures, had more interest in the 
subject matter, were less intimidated by research reports, grew in ability to comprehend 
research, and felt pride in realizing they could think at this level. 

This application of APRR seems to be a useful and engaging way to teach in a freshman 
seminar. Lessons learned include the need to provide more advance explanation about the 
nature of research in the field and more professor feedback on the students’ simulated 
peer reviews. But, the results do lend support to the claim of others that APRRs can teach 
the nature of scholarship better than textbooks. 

Keywords: Adapted research reports, seminars, teaching scholarship, peer review. 


This study was conducted in a one credit-hour 15-week freshman seminar course on 
Learning and Memory in which students accessed the core academic content in the form 
of adapted published research reports (APRRs).The purpose was to increase incorpora¬ 
tion of scientific thinking into traditional curricula in an attempt to enrich traditional 
teaching and help students develop capability in critical and creative thinking and prob- 
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lem solving, The use of APRRs is a relatively new idea in science education and appar¬ 
ently has not been used as done here in a seminar context nor has anyone seemed to have 
required students to work as analysis teams to conduct simulated peer reviews. Specific 
hypotheses about the APRR teaching approach tested here include: 

1. Content mastery: this kind of supplementary teaching should promote greater 
student engagement with the content and more active learning than with lectur¬ 
ing. Students should learn about the research process. 

2. Self assurance: students should improve their ability to evaluate research, feel 
more confident in analyzing research reports, and be positively reinforced by the 
realization that they could generate insights that were not presented by profes¬ 
sional authors. 

The purpose of this report is to show how this approach to teaching can be relevant for 
any one-hour seminar course, while providing some insight on how students might react 
if APRR experience were incorporated into a traditional beginning science or engineering 
course. By testing with freshmen, findings might be applied to gateway courses in sci¬ 
ence in the hope of improving their learning in later courses, reducing attrition, and at¬ 
tracting new science majors. 

To promote intellectual engagement with the content of APRRs, students were required 
first individually and then as analysis teams to conduct simulated peer reviews and pre¬ 
sent their analyses to the class. 

Rationale 

Introductory science courses are often the target of the common criticism of undergradu¬ 
ate science and technology education as too broad and shallow (National Research Coun¬ 
cil, 2011). Many critics of traditional science and technology education charge that tradi¬ 
tional lectures tend to bore students and lack proper emphasis on scientific thinking and 
process (Aldridge. 2012). The NSF report, “Shaping the Future: New Expectations for 
Undergraduate Education” (NSF, 1996) stresses the need to develop critical thinking 
skills, address the current applications and implications of didactic instruction, and en¬ 
courage students to develop skill in communicating scientific ideas and thinking. 

Science educators increasingly realize that traditional education in introductory courses 
does not give students enough experience with the essence of science and technology: 
scientific reasoning and argumentation (Llewellyn & Rajesh, 2011). Students frequently 
fail to see how knowledge is constructed and they misinterpret presentation of science in 
popular media and everyday conversation (Cavagnetto, 2011). Students likewise do not 
appreciate the central and crucial role played by reading and writing in the advancement 
of science. Their focus too often is on memorizing enough material to pass multiple- 
choice examinations. 

Undergraduate students should learn to think like scientists in order to realize the excite¬ 
ment and joy of discovery. Using scientific journal articles to teach scientific inquiry and 
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reasoning skills lets students obtain a deeper and more authentic understanding of the sci¬ 
entific method and processes than is provided in other teaching resources. Scientists learn 
their craft largely from reading journal articles. Undergraduate students could learn in the 
same way if the journal articles were adapted for easier comprehensibility. 

Science, as practiced by professionals, requires peer review by outside experts who cri¬ 
tique the experimental design, methods, data, and the scientific reasoning the authors 
used to construct their hypotheses, explanations, and argumentation. Why can’t students 
learn about science the way it is practiced in the real world? 

What might APRR learning experiences provide that is not already readily available in 
many traditional lectures and textbooks? APRR learning activities can address many of 
the key findings about how science is learned. APRRs provide the specific advantage of 
focus on core ideas and minimize jargon and emphasis on terms. Scientific communica¬ 
tion seems increasingly “overloaded with unnecessary information, technical detail, and 
so cluttered with abbreviations, jargon, and acronyms as to be nearly incomprehensible to 
anyone but the specialist” (Schatz, 2012). 

Osborne (2009) suggests that science has become too advanced for lay persons, specifi¬ 
cally beginning college students, even if research papers are adapted to be more under¬ 
standable. Osborne claims that the “entry cost” for the novice reader becomes higher and 
higher with each ensuing generation. If true, science will become too complicated for 
students, and science could enter a dark age where there are only a few scientists left. Os¬ 
borne did, however, stress that teachers often misrepresent science as a “hands-on” activ¬ 
ity (of pushing buttons on a machine, doing manipulations in a fume hood, or whatever), 
instead of being more of a “minds-on” activity—which is the main point of APRR teach¬ 
ing. Thus a key objective of the present study was to test the notion that research reports 
are beyond the understanding of the novice, in this case freshmen college students. 

Prior APRR Studies 

The idea of teaching with APRR fits well with the current educational emphasis of stress¬ 
ing “inquiry learning.” APRRs emphasize the inquiry nature and the value of evaluating 
research papers as a way to learn science (Yarden, 2009). In an isolated use of an adapted 
paper on a mathematical model for West Nile virus epidemiology, Norris, Macnab, Won- 
ham, & de Vries (2009) report that the APRR experience helped high-school students 
correct some common misconceptions about mathematical models. 

Two valuable features of APRRs, authenticity and support for teachers, were noted by 
Yarden et al. (2009). They claimed that APRRs can promote the learning of science con¬ 
tent as well as science process and that APRRs expose students to an unfamiliar genre of 
scientific writing. 

The approaches of which this author is aware differ from the application in this present 
study in several key ways. The present study used complete research reports, not just the 
abstract and introduction of a research report as used by Falk and Yarden (2011). Ira 
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Clark and colleagues (Clark et al. 2009) used a form of APRR in a seminar environment 
in which the professors provided lecture explanation. This approach eliminates the need 
(and benefits) for students to read (which today’s students often avoid when possible). 

None of the studies of which I am aware required students either to 1) conduct a critical 
analysis in the form of simulated peer review, or 2) create an academic deliverable such 
as a written report or class presentation on the analysis. 


Methods 

Students 

APRR activities were used in a Freshman Year Seminar class on learning and memory 
given in the Fall of 2012 for incoming students. The course met for one hour once a 
week, with pass/fail grading and a one-semester credit hour credit. Attendance was man¬ 
datory. Enrollment consisted of twenty four students in 13 academic majors. 

Adapting Published Research Reports 

Eight published research reports on learning and memory were re-written by the author as 
a third-party description of what was said in the original, only with simplification, foot¬ 
notes of added explanation, condensation of some data, and elimination of references 
and the more arcane or tangential parts of the original. Each adapted paper maintained the 
structure and overall style of the original report but was adjusted to the conceptual under¬ 
standing, reading level, vocabulary, and mathematical skill of undergraduate college stu¬ 
dents. Ideas in the original text were not enhanced, nor were professor opinions inter¬ 
jected. The reading level was word-processor scored as 12th grade or less. After adapta¬ 
tion, the page length was usually reduced to 3 to 5 pages. 

Four APRRs were assigned in the first half of the semester and four in the second half. 
Although a given student group only analyzed one of each set of four APRRs in depth, 
they were required to have a minimal knowledge of the other three papers so that they 
could participate more meaningfully in debate and discussion over the conclusions 
reached when the in-depth analysis of those papers was presented by other students. 

Simulated Peer Review 

Each student individually answered 21 questions intended to serve as scaffolding for the 
analysis (see Appendix). Questions stressed the need for critical thinking and insight. 
Each student had to show the professor a bullet-list response to each of the scaffolding 
questions on the paper they were assigned for in-depth analysis. This assured that each 
student had thought about the issues and was prepared to contribute to the group analysis. 
Then, I formed four heterogeneous analysis teams of six members each, which were to 
use some of the formalisms of effective collaborative learning (Gabbert, Johnson, & 
Johnson, 1986; Johnson, Skon, & Johnson, 1980; Johnson & Johnson, 1981; Johnson & 
Johnson, 1989; Kadel & Keehner, 1994). In general, the educational literature shows that 
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collaborating students are more active learners than they would be from passively listen¬ 
ing to lectures. They improve their understanding and insight about content from group 
interaction. They should also learn communication and small-group “struggle and sub¬ 
tlety” social skills that are relevant to a laboratory or office after graduation. 

Each analysis team selected its own team captain, and the other members negotiated role 
assignments. Teams had four weeks to debate the issues, reach consensus, and develop 
their simulated peer review presentation to the class. 

Students were cautioned to avoid trivial comments and reminded that no one study is 
supposed to be exhaustive. Students were to think of a research report as a package of 
ideas. The issue for them to emphasize was not so much the size of the package but the 
quality of its contents. Instructions urged students to focus on matters of theoretical con¬ 
cepts, design, methodology, and interpretation that the investigators could have improved 
on had they thought hard enough about it. Students were encouraged to be innovative and 
challenge weaknesses in the paper. Students were reminded of the obligation of every 
student in the group to participate in the consensus building on answers to the scaffolding 
questions. The analysis was to emphasize insight on ideas and conclusions that were ap¬ 
parently not recognized by the authors. 

Classroom Implementation 

The basic approach was tested in a First Year Seminar class, a recently established pro¬ 
gram of the university aimed at helping students adjust to the intimidating environment of 
this huge (50,000+ on-campus enrollment) research university. The idea is to enroll in¬ 
coming first-year students in a seminar of their choice in a small-class, low-stakes envi¬ 
ronment (maximum of 20 students). The enrollment limit for the learning and memory 
seminar described here was raised to 25 because of the course popularity (one student 
dropped the course after the first class meeting). 

These seminars are supported by many senior faculty, and it provides a way for incoming 
students to become acquainted with the university’s top-ranking professors. Teaching 
such seminars seems to appeal to senior professors because it is an easy task for them and 
the environment is more informal and cordial for both students and professors than in 
typical high-stakes courses. At the professors’ discretion, the seminars are graded or 
listed as pass/fail. 

Class periods are 50-min long, and in this Learning and Memory seminar usually in¬ 
volved student work on the APRRs, followed by 15-20 minute lectures. In the first two 
class periods students worked alone a single APRR assigned for their simulated peer re¬ 
view without knowing who else in the class was working on the same APRR. 

All students had to pass a quiz on the three APRRs that were not designated for in-depth 
analysis. A short lecture explained that research papers have in principal just four parts: 
WHY (rationale and hypotheses for performing the study, which is presented in the pa¬ 
per’s Introduction); HOW (experimental design and methods, found in the Methods sec- 
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tion), WHAT (the basic findings, found in the Results section), and SO WHAT (the im¬ 
plications and “take-home” message of the paper, found in the Discussion section). For 
each category, students were told it would suffice to memorize one or a few bullet points 
that captured the essence of each part of the paper. 

To make memorization of the bullet points easier, students were taught the well-known 
memory-peg technique in which each the ideas was represented in an image that inte¬ 
grated the bullet-point ideas in an imagined scenario or story. Students were taught on the 
first day of class a new composite flash-card study system (Klemm, 2012a). In this sys¬ 
tem, clip-art images are pasted in table format in an animated PowerPoint, one row of 
four icons for each unassigned report (memory of information is facilitated by where it is 
located and images are easier to remember than words). Self-testing during study was to 
consist of anticipating what the first icon represented (WHY) and how that image served 
as a mnemonic peg for the associated bullet points. Then, a mouse click advanced to the 
next icon, and so on. Readers can download a sample composite flash card from a link at 
http: //thankyoubr ain. com/consult ant. htm . 

Then, in the third class period, heterogeneous teams were formed and students began 
team building and group consensus throughout periods 4-6 and took a quiz on the three 
unassigned reports. In the 7 th and 8 lh class periods, student groups gave their simulated 
peer review presentations to the class and conducted class discussion. Two teams gave 
presentations in each class period. 

In the second half of the semester teams remained intact and four new APRRs were used. 
We repeated processes of the first half, with a couple of exceptions: 1) All team members 
switched roles, so that each student gained experience with at least two team roles; 2) For 
the three papers that were not assigned for simulated peer review, each student individu¬ 
ally and independently prepared a composite flash card mnemonic for the Why, How, 
What, and So What aspects of the reports. This was to be in PowerPoint (not Word, as 
many students had used in the first half of the semester-see Results) so they could self¬ 
test in flash-card mode and discover how much easier things are to memorize that way. 

Tests of Hypotheses 

Content mastery and confidence were assessed by an end-of-course survey that measured 
student self-assessment of content mastery and confidence in analyzing research reports. 
Students completed the post-course survey (10 questions on a 5-point Likert scale) on the 
last day of class. To prevent mindless box checking, some of the questions were worded 
negatively, and students were advised of the need to read questions carefully. Six of the 
questions dealt specifically with the simulated peer review experience, and only those 
results are reported here. 
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Results 

Quiz performance on Unassigned Papers 

On the quizzing for the first half of the semester, only two of the 24 students got all the 
answers correct for all three of their unassigned papers. It was clear that the vast majority 
of students had recall difficulties, even though they had three weeks to prepare and mem¬ 
orize simple bullet-point answers for the “Why, How, What, and So What” of three pa¬ 
pers. Because it seemed clear that most students did not use the mnemonic strategies that 
had been covered in the lectures and in their text (Klemm, 2012b), students had to show 
evidence for the next set of APRRs that they had developed a memorization strategy by 
using the composite flash card system (Klemm, 2012a). However, when students submit¬ 
ted their “flash card,” many students prepared them in a MS Word document, not Power¬ 
Point. That is, the icons and associated bullet lists were pasted into one document without 
any animation. In that form, students fell back into the old study habit of “looking over” 
material to be memorized, not using the power of mental images and explicit self-testing. 
There was no way to know if they used the images as mnemonic pegs, but the poor test 
results suggested they did not. 

Group Simulated Peer Review Presentations 

Students were told to go beyond simple bullet-point slide show format that listed their 
answers to the scaffolding questions. Even in the first set of presentations, the students 
presented their analyses in an engaging and informative way. All groups effectively used 
either two or three team members to deliver the presentation. The slideshows were well 
crafted. One group even used Prezi instead of PowerPoint. However, the depth of analy¬ 
sis and insight were not at the hoped-for level. Most students were not rigorous or crea¬ 
tive analysts. 

Student Self-assessment 

Content Mastery. More than 2/3 of the class (score of 4 or 5) believed they learned more 
about the subject of the two papers they analyzed than they would have if that material 
had been presented as lecture (Fig. 1). 

Only about 1/3 of the class believed they learned much about the research process, 
while nearly 45% reported no effect (Fig.2). About half the class thought the simulated 
peer review was more interesting than listening to lectures, but the others concluded there 
was either no difference or else preferred lectures (Fig. 3). 

Confidence. Nearly half the class reported feeling less intimidated by research reports 
after the APRR experience. The other half (approximately) felt the same as before. (See 
Fig. 4.) 
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Figure 1. Responses to: “I will remem¬ 
ber more than if I heard it in lecture.” 



Figure 2. Responses to: “I don’t think 
I learned much about research proc¬ 
ess.” 



Figure 3. Responses to: “Peer review was 
more interesting than lectures.” 



less intimidated by research reports 
than before.” 


Over 2/3 of the class believed they were better able to analyze research papers than be¬ 
fore this experience (Fig. 5). No one felt less able in this respect. Over 2/3 of the class 
were pleased to realize that they generated ideas that did not seem to occur to the profes¬ 
sionals (Fig. 6). 


Discussion 

Student Learning Strategies. Resistance to change in memorization strategy seemed 
marked. For the first quiz on the three non-assigned reports, most students clearly were 
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Figure 5. Responses to: “I can analyze 
experiments better than before.” 



Figure 6. Responses to: “It feels good to 
get ideas that did not seem to occur to 
authors.” 


using the rote-memorization strategy they had used during their previous 12 years of 
schooling. This was evident from the poor test scores and from the fact that many stu¬ 
dents did not use the PowerPoint format that would have allowed flash-card style self¬ 
testing. Informal surveys of these students and others outside of this course indicate that 
many students have no learning strategy, relying on repeated “looking over” the material 
until they think they have learned enough to pass tests. The idea of using mental imaging 
mnemonics is apparently alien. Breaking well-ingrained habits of learning is apparently 
difficult to achieve in a short time. 

The level of critical and creative thinking seemed disappointing to this professor, sug¬ 
gesting that these skills were not well developed in secondary school. To the extent that 
this conclusion is correct and generalizable, it suggests that a price is being paid for 
school emphasis on learning expected answers to state-standard questions. This, of 
course, adds justification to making this kind of experience available early in a college 
career. 

In fact, why not provide some APRR experience in secondary school? Such teaching 
seems ideal for helping teachers meet the expectations of the National Research Coun¬ 
cil’s new Framework for K-12 Science Education (NRC, 2011). Simulated peer review of 
adapted reports specifically addresses the Framework's requirements for students to learn 
how to 1) ask questions and define problems, 2) analyze and interpret data, 3) construct 
explanations and design solutions, 4) engage in argument from evidence, and 5) obtain, 
evaluate, and communicate information. 

Student Self-assessment 

A strikingly large number of students thought they could learn more from research report 
analysis than if they were presented the same didactic content via lecture (Fig.l). How¬ 
ever, this conclusion is tentative, in that no formal examinations were used to compare 
how much material is retained under the present condition versus lecture instruction on 
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the same material. Despite this limitation, much memory research clearly establishes that 
thinking about academic content is profound memory rehearsal and one of the best ways 
to memorize—clearly superior to rote memorization (Klemm, 2012c). 

Since most students did not think they learned much about research process (Fig. 2), there 
may be a need for a few introductory lectures on how research is done before launching 
into journal article analysis. Also, more in-depth debriefing was called for, yet was not 
performed because the class schedule called for two student-group presentations in one 
50-minute period. More time needs to be made available for the professor to critique stu¬ 
dent presentations, including explanation of the experimental design and methods used in 
a particular report. This conclusion is buttressed by the finding that about half the class 
gained little confidence in analyzing research (Fig. 4). Even so, the others did gain confi¬ 
dence, in spite of analyzing only two papers under less than optimal conditions. 

About half the class retained their comfort for listening to lectures (Fig. 3). Perhaps this 
reflects their long history of receiving instruction via lecture format. On the other hand, 
since half the class preferred this APRR experience over lecturing, perhaps the others 
would change their preference once they had more exposure to teaching that required 
more active learning than listening to lectures. 

Both measures of student confidence (Figs. 5 and 6) showed large majorities had a new 
appreciation for their capacity for creative and critical thinking. While this professor was 
not particularly impressed with the rigor of their thinking or level of creativity, it is im¬ 
portant for students to develop the confidence that they can get better at it. Moreover, had 
debriefing sessions been more thorough, it is reasonable to expect that students would in 
fact learn to be more rigorous in their analysis and more insightful. We need to remember 
also that these are students fresh out of high school in their first semester of college. This 
kind of experience early-on in a college career might help students gain more out of later 
courses. 

Comparison with Other APRR Studies 

This present study differed from previous reports of APRR experiments in that the stu¬ 
dents operated as a formally structured analysis team. Yet prior to group work, each stu¬ 
dent had to document individual responses to the scaffolding questions provided to guide 
analysis. Moreover, the analysis teams had to submit a tangible deliverable (class presen¬ 
tation) rather than just participate in casual class discussion. 

This teaching approach enabled coverage of eight adapted research papers and multiple 
mini-lectures in a one credit-hour seminar format. Thus, a seminar course on any topic 
could be conducted similarly with research papers selected to fit the seminar topic. This 
same plan could be modified as part of a traditional four credit hour course, perhaps as a 
substitute for some of the traditional lecture hours or recitation sections. 

The results of improved student comfort, confidence, and interest in learning from re¬ 
search reports is consistent with similar findings by Hoskins, Lopatto, & Stevens (2011). 
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They found that upper division and graduate students who were systematically guided 
through review of non-adapted research publications developed improved attitudes about 
the nature of science. Their post-course surveys revealed significant changes in students’ 
self-assessed confidence in their ability to read and analyze primary literature, under¬ 
standing of the nature of science, and epistemological beliefs (e.g., their sense of whether 
knowledge is certain and scientific talent innate). 

Other Applications for APRR Teaching 

Professors in multiple STEM disciplines would ordinarily use APRR exercises as a 
change-of-pace supplement to their traditional teaching—perhaps to augment a particu¬ 
larly difficult or important component of the course. Options for deploying APRR activi¬ 
ties include: 1) inserting them as homework after lectures have provided appropriate 
background, 2) substituting them for term papers, 3) substituting them for one or more 
lab or recitation activities, and 4) making them the core of an entire course, such as a 
seminar or honors course. Another option is for the professor to introduce the main issue 
in the APRR, ask students how they would address it, and then use the APRR to illustrate 
how it was actually addressed. Yet another option is a Citation Index exercise, wherein 
students create citation maps of the papers that cited the APRRs and explain how the cit¬ 
ing papers relate to each other (Klemm, 1976. See also Klemm, 2009). 

Professors could integrate popular media reports on the APRR, both to pique student in¬ 
terest and to present basic concepts before an APRR learning activity. Including popular 
media reports is also important because after graduation most students will receive sci¬ 
ence information mainly from popular media sources (television, newspapers, websites, 
etc.), and their APRR experiences will help them evaluate the content more meaning¬ 
fully. Students without APRR experiences interpret popular media reports quite differ¬ 
ently than experts (Zimmerman, Bizanz, & Bizanz, 1999). 

These results suggest a way to enrich gateway courses with supplementary learning mate¬ 
rials that are more engaging and authentic than typically found in lectures and textbooks. 
Expected benefits include: 

1. More active learning, 

2. A way to provide inquiry learning experiences without the cost and prepara¬ 
tion needed in wet labs, 

3. A stimulating change-of-pace from traditional classes, 

4. An opportunity for developing critical and creative thinking skills, 

5. Increased long-term student interest in specific research issues, student confi¬ 
dence building, and opportunities to recruit students to become college majors 
in science and engineering disciplines. 

My impression, though not systematically measured, was that these APRR experiences 
accelerated the transition from high-school level learning styles to the more mature ap¬ 
proach needed in college. The right choice and application of APRR activities in a course 
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could promote an academic culture that engages and inspires students, as well as prepares 
them for more mature approaches to learning in their later classes. 

Finally, the APRR approach seems scalable. Certainly, the experience reported here indi¬ 
cates the suitability for small seminar classes. But even with hundreds or thousands of 
students in a class, direct professor-student interaction is simplified by the “command 
structure” of analysis-team captains and the systematic division of labor provided by 
team member roles. Large-class lectures can be replaced with didactic content presented 
on-line as videos, readings, and slide shows, with the complement of APRR as problem- 
based learning experiences that provide in-depth coverage of a topic, help develop col¬ 
laboration and communication skills. 


Conclusions 

APRR activities help students focus on the issue of how we know and the nature of evi¬ 
dence. Paradoxically, argument and debate are common in science, but are virtually ab¬ 
sent from science education (Osborne, 2010). Moreover, in this present implementation 
of research report analysis, the learning is spaced over time so students have time to find 
related information, ruminate, and debate among themselves. APRRs can teach how: 1) 
science and technology issues are identified and approached for testing; 2) ideas evolve 
from historical perspectives; 3) various experimental designs and methods have advan¬ 
tages and limitations; 4) data are illustrated and evaluated statistically; 5) experiments can 
test hypotheses (that is, the difference between data and evidence); 6) research relates to 
traditional course content; and 7) research is a necessary element of university education. 

The simulated peer review of APRR gives students the opportunity to develop their ca¬ 
pacity for insightfulness, helping students learn to think critically and creatively, not just 
memorize answers for multiple-choice exams. Knowing the WHAT of a given discipline 
is not enough. Students also need to know the WHY, HOW, AND SO WHAT. In-depth 
analysis of APRRs can enhance such skills such as questioning, predicting, connecting to 
prior knowledge, and summarizing. APRRs promote critical thinking skills such as per¬ 
suasion, interpretation, consideration of multiple perspectives, evaluation, and applica¬ 
tion. 

This application of APRR seems to be a useful and engaging way to teach a freshman 
seminar. Lessons learned include the need to provide more advance explanation about the 
nature of research in the field and more professor feedback on the students’ simulated 
peer review. The simulated peer review approach can accomplish something that is often 
neglected in introductory science and technology courses: require students to show in¬ 
sight. Students can develop insight capability if it is expected of them, but this present 
study revealed that beginning students have not generally developed capacity for creative 
scholarly thought. 

Nonetheless, this approach has the rare benefit of engaging students at every level of 
Bloom’s Taxonomy of Learning Domains (Lightle, 2011). Moreover, the results do lend 


The Journal of Effective Teaching, Vol. 13, No. 2, 2013, 6-20 

®2013 All rights reserved 



Klemm 


18 


some support to the claim of others that research reports can teach the nature of scholar¬ 
ship better than textbooks (McComas, Clough, & Almazroa, 1998). 
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Appendix. Research Paper Analysis Questions 


Introduction. 

1. Was there an explicit hypothesis? If not, what was the implicit hypothesis? 

2. How reasonable does the rationale seem? Why or why not? 

3. What are some alternative ideas that were not considered. Does this research seem 
scientifically important? Is it important in other ways? Why or why not? 

Methods. 

1. Is the design adequate? Why or Why not? 

2. How well do the control groups serve as checks on variables that could influence 
results other than what is being tested? Why or why not? 

3. Describe the negative control group and its function? Are there important vari¬ 
ables that the control group does not account for? 

4. Is there a positive control group or is one needed? 

5. Is double-blind testing needed and used? Why or why not? 
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6. Do the data-collecting approaches or devices seem appropriate? Are they sensitive 
enough for what is being tested? 

7. Are there other approaches or devices that might have been better to use? 

Results 

1. Do the results support the hypothesis or not? How convincing is that support? 

2. Do you notice anything of potential importance in the data that was not com¬ 
mented on by the authors? 

3. Is the variance in data large enough to suggest that some variables are not being 
controlled? What might these be? 

4. Apart from the statistical effect, what is the magnitude of the ‘treatment’ effect? Is 
it large enough to be of much practical importance? 

Discussion 

1. Summarize how the authors discussed the results in terms of their original hy¬ 
pothesis. 

2. Did they point out implications that go beyond the hypothesis? 

3. What implications did the authors perceive that go beyond the original hypothesis. 
Do you perceive any other implications? 

4. What ideas for future research did the authors generate? What ideas for future re¬ 
search do you generate? 

5. Note any important information that was not commented on by the authors. 

6. Does the author state a “take-home” lesson? 

7. How would you state the “take-home” lesson? 
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